Advancements in image-processing techniques including new processes and the use of more powerful graphics processors has lead to increased reliance on visual data. For example, streams of 2D images, typically referred to as videos, are analysed and a 3D model is generated therefrom. The generation of 3D models is used in the entertainment, gaming, natural sciences, medicine, inspection, reverse engineering, and mining industries. Some 3D models are generated to produce a ‘virtual’ world—an artificial 3D image in an artificial world as is used in animation or gaming; these 3D models need not be accurate and, in fact, are often modified afterwards to improve the overall effect. In other applications, 3D models are generated from 2D photographs or videos captured of a real world object. Also 3D models are formable by scanning a real world object with a 3D scanner. These 3D models provide valuable information of real world objects in a compact format easily understood by human beings.
Range information provides a distance measure between a measurement apparatus and a target. Typically, a range between a range sensor and a point in 3D space is determined. For generating a 3D model of a real world object, multiple ranges are determined to form a 3D map of the closest surface(s) to the range sensor. For example, a distance sensor faces a wall, the point on the wall directly in front of the distance sensor is the closest point to the distance sensor and is also considered to be the center point. Points on the wall further from the center point are also further away from the distance sensor. Ranges measured between points on the surface of a real world object in 3D space aid in determining the shape of the surface in the corresponding 3D model. For example, if the wall is flat, the distance between any two adjacent points on the wall is an expected value. However, if the wall is not flat, the distance between two adjacent points on the wall is other than the expected value, indicating that the wall is not flat, but instead is curved, or is at least curved at that portion of the surface of the real world object. There are many range sensors including, but not limited to, Lidar, infrared (IR), Sonar, stereoscopic, LASER, ultrasound, patterned light, RF, etc. that are used in many different applications. That said, essentially, each range sensor attempts to determine a distance between the range sensor and a target.
When a 3D model is generated using 3D imaging, the range information sensed or calculated is viewed as source data and is either archived or discarded, depending on the application. Thus, when a range sensor is used to scan a space for forming a 3D map thereof, the range values are not part of the 3D model, though the 3D model is based on them. Once the 3D model is generated, the range data has been utilised for its intended purpose. Similarly, with stereoscopic range finders, once the 3D model is generated, the image data has served its purpose. Of course, in stereoscopic 3D modeling, it is known to superimpose a skin on the 3D model to provide it with color, texture, etc. allowing for rendering of image data from the 3D model.
In auto-industry quality assurance applications, 3D models are constructed from 2D videos of newly assembled parts of a car. Some car parts are complex to assemble and have tight tolerances. Once 3D models are constructed from the 2D images, the 3D model of the car part is compared to an original computer aided drawing (CAD) model to determine if the assembled car part meets engineering standards. In this application, the source data is often discarded.
For industries requiring inspection of a real world object in an installed state, including both animate and inanimate objects, it is known to capture video footage of the installed object at intervals, usually by an operator, and to review the object for irregularities, potential problems, and wear by experts in the field. The operators are often trained to work in extreme environments, such as in mines, underwater, or on high towers. Alternatively, operators are trained to use complex inspection equipment such as unmanned remote operated vehicles (ROVs) or robots. In some applications, the operator annotates the video footage when the operator notices a condition or situation of note while recording the video, for example, by recording a time indicator for the video and a brief description of the issue found. The video when reviewed by industry experts, is either more closely analysed or analysed only before, during and after the time noted by the operator to determine if there indeed is an issue. If an issue is determined, the expert decides what action is required or if the noted problem should be monitored over time. Without annotation by the operator, the experts are forced to view carefully hours of video footage to detect an issue with the object onscreen. Alternatively, the experts monitor live videos feeds as video data is captured.
It is difficult to automatically associate captured video data with items of interest, areas of concern, etc. without manually indicating such for the expert reviewer or without navigating to those areas of interest by the expert themselves.
It would be advantageous to overcome some of the disadvantages of the prior art.